WORD REPRESENTATIONS

Advances in Pre-Training Distributed Word Representations (L18-1)

Challenge:	Pre-trained language models (PLMs) construct word representations at subword level with Byte-Pair Encoding (BPE) or its variations . but these methods split a word into subword units and make it incomplete and fragile .
Approach:	They propose a character-aware pre-trained language model to tackle OOV problems . they construct contextual word embedding for each token from sequential character representations .
Outcome:	The proposed model improves on the existing models on multiple NLP benchmarks.

Copied to clipboard

Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, Armand Joulin

DebIE: A Platform for Implicit and Explicit Debiasing of Word Embedding Spaces (2021.eacl-demos)

Challenge:	Pre-trained word representations are a building block of many Natural Language Processing and Machine Learning applications.
Approach:	They propose to combine known tricks and a set of publicly available pre-trained word vector representations to train high-quality representations.
Outcome:	The proposed models outperform the current state of the art on a number of tasks while maintaining a high training speed to scale to massive amount of data.

Copied to clipboard

Niklas Friedrich, Anne Lauscher, Simone Paolo Ponzetto, Goran Glavaš

Responsible & Trustworthy NLP

Challenge:	Recent research has shown that distributional word vector spaces often encode stereotypical human biases, such as racism and sexism.
Approach:	They propose a platform that measures and mitigates bias in word embeddings by executing two (mutually composable) debiasing models.
Outcome:	The proposed platform can measure and mitiga bias in word embeddings.

Joint Semantic and Distributional Word Representations with Multi-Graph Embeddings (D19-53)

Copied to clipboard

Pierre Daix-Moreux, Matthias Gallé

Representing ELMo embeddings as two-dimensional text online (2021.eacl-demos)

Challenge:	Prior work has shown that word embeddings can be improved by using semantic knowledge-bases.
Approach:	They propose a way to combine distributional and semantic information while preserving lexical information of co-occurrences of words.
Outcome:	The proposed method improves word embeddings on a variety of word similarities.

Copied to clipboard

Andrey Kutuzov, Elizaveta Kuzmenko

Domain-Specific Word Embeddings with Structure Prediction (2023.tacl-1)

Challenge:	ELMoViz module adds support for contextualized embedding architectures, in particular for token embeddable word models.
Approach:	They propose to add a module to the free and open-source WebVectors toolkit which provides lexical hyperlinks to word representations in static embedding models.
Outcome:	The ELMoViz module adds support for contextualized embedding architectures, in particular for ELMa models.

Copied to clipboard

David Lassner, Stephanie Brandl, Anne Baillot, Shinichi Nakajima

LINSPECTOR WEB: A Multilingual Probing Suite for Word Representations (D19-3)

Challenge:	Current word embedding methods do not provide a way to use or predict information on structure between sub-corpora, time or domain.
Approach:	They propose a word embedding method that provides general word representations for the whole corpus, domain-specific representations and embeddable alignment simultaneously.
Outcome:	The proposed method provides better performance than baselines on a dataset of science and philosophy articles.

Copied to clipboard

Max Eichler, Gözde Gül Şahin, Iryna Gurevych

Compound or Term Features? Analyzing Salience in Predicting the Difficulty of German Noun Compounds across Domains (2021.starsem-1)

Challenge:	LINSPECTOR WEB is an open source multilingual inspector to analyze word embeddings.
Approach:	They propose to use LINSPECTOR WEB to analyze word embeddings in 28 languages.
Outcome:	The system performs 16 simple linguistic probing tasks for a diverse set of 28 languages.

Copied to clipboard

Anna Hätty, Julia Bettinger, Michael Dorna, Jonas Kuhn, Sabine Schulte im Walde

Challenge:	Using domain-specific vocabulary, it is important to analyse domain-related characteristics to improve the communication between lay people and experts.
Approach:	They focus on the interaction of compound-based lexical features (such as frequency and productivity) and terminology-based features (contrasting domain-specific and general language) across word representations and classifiers.
Outcome:	The proposed model shows that the interaction of compound-based lexical features and terminology-based features across word representations and classifiers is important for a broad binary distinction into ‘easy’ vs. ‘difficult’ general-language compound frequency is sufficient, but for . a more fine-grained four-class distinction it is crucial to include contrastive termhood features and compound and constituent features.

Distilling Linguistic Context for Language Model Compression (2021.emnlp-main)

Copied to clipboard

Geondo Park, Gyeongman Kim, Eunho Yang

Can Network Embedding of Distributional Thesaurus Be Combined with Word Vectors for Better Representation? (N18-1)

Challenge:	Knowledge distillation is a major technique for deploying vast language models in resource-strapped environments.
Approach:	They propose a method that transfers contextual knowledge via Word Relation and Layer Transforming Relation.
Outcome:	The proposed method is able to transfer contextual knowledge without restrictions on architectural changes between teacher and student on language understanding tasks.

Copied to clipboard

Abhik Jana, Pawan Goyal

Solving Data Sparsity for Aspect Based Sentiment Analysis Using Cross-Linguality and Multi-Linguality (N18-1)

Challenge:	Distributed representations of words learned from text have proved to be successful in various natural language processing tasks.
Approach:	They propose to embed a distributional thesaurus network into dense word vectors and compare them to state-of-the-art word representations.
Outcome:	The proposed representations improve performance against state-of-the-art word representations even without handcrafted lexical resources.

Copied to clipboard

Md Shad Akhtar, Palaash Sawant, Sukanta Sen, Asif Ekbal, Pushpak Bhattacharyya

Responsible & Trustworthy NLP

Challenge:	Efficient word representations play an important role in solving various problems related to NLP, data mining, text mining etc.
Approach:	They propose to leverage bilingual word embeddings learned through a parallel corpus to minimize the effect of data sparsity.
Outcome:	The proposed model is tested against state-of-the-art methods in two experimental setups.

Self-supervised Post-processing Method to Enrich Pretrained Word Vectors (2023.findings-emnlp)

Copied to clipboard

Hwiyeol Jo

Continuous Learning in Neural Machine Translation using Bilingual Dictionaries (2021.eacl-main)

Challenge:	Existing methods that use external resources to make word vectors specialize depend on the lexicon.
Approach:	They propose a self-supervised extension of extrofitting by its own word vector distribution.
Outcome:	The proposed method improves word similarity embeddings on similarity tasks without external resources.

Copied to clipboard

Jan Niehues

Explaining Word Embeddings via Disentangled Representation (2020.aacl-main)

Challenge:	Recent advances in neural machine translation have led to astonishing translation quality of research systems.
Approach:	They propose to integrate one-shot learning methods with different word representations to assess the ability of neural machine translation to continuously learn new phrases.
Outcome:	The proposed framework improves translation quality of bilingual dictionaries from 30% to 70%.

Copied to clipboard

Keng-Te Liao, Cheng-Syuan Lee, Zhong-Yu Huang, Shou-de Lin

Illustrative Language Understanding: Large-Scale Visual Grounding with Image Search (P18-1)

Challenge:	Disentangled representations are known to represent interpretable factors in separated dimensions.
Approach:	They propose to transform dense word vectors into disentangled embeddings with improved interpretability by encoding polysemous semantics separately.
Outcome:	The proposed model can be encoded into multiple sub-embeddings or sub-areas and generates more efficient and effective features for natural language processing.

Copied to clipboard

Jamie Kiros, William Chan, Geoffrey Hinton

Visual Data in NLP

Challenge:	a large-scale lookup operation to ground language via ‘snapshots’ of our physical world accessed through image search is currently used to learn word representations.
Approach:	They propose a large-scale lookup operation to ground language via ‘snapshots’ of our physical world accessed through image search.
Outcome:	The proposed model is based on a large-scale lookup operation to ground language using image search.

Transformation Networks for Target-Oriented Sentiment Classification (P18-1)

Copied to clipboard

Xin Li, Lidong Bing, Wai Lam, Bei Shi

Text Classification

Challenge:	a new model for sentiment classification uses attention instead of attention to classify sentiment polarities over individual opinion targets.
Approach:	They propose a model that uses a CNN layer to extract salient features from transformed word representations from a bi-directional RNN layer.
Outcome:	The proposed model achieves state-of-the-art on a few benchmarks.

Robust Multilingual Part-of-Speech Tagging via Adversarial Training (N18-1)

Copied to clipboard

Michihiro Yasunaga, Jungo Kasai, Dragomir Radev

Syntactic Text Processing

Challenge:	Adversarial training (AT) is a powerful regularization method for neural networks, aiming to achieve robustness to input perturbations.
Approach:	They propose and analyze a neural POS tagging model that exploits adversarial training by training on unmodified and adversarials.
Outcome:	The proposed model improves overall tagging accuracy and prevents over-fitting in low resource languages and boosts tabbing accuracy for rare / unseen words.

A Systematic Study of Leveraging Subword Information for Learning Word Representations (N19-1)

Copied to clipboard

Yi Zhu, Ivan Vulić, Anna Korhonen

Tiny Word Embeddings Using Globally Informed Reconstruction (2020.coling-main)

Challenge:	Existing word representation models for morphologically rich languages use subword-level information, but their systematic comparative analysis across typologically diverse languages and tasks is still missing.
Approach:	They propose a framework for learning subword-informed word representations that allows for easy experimentation with different segmentation and composition components.
Outcome:	The proposed framework allows for easy experimentation with different segmentation and composition components, as well as advanced techniques based on position embeddings and self-attention.

Copied to clipboard

Sora Ohashi, Mao Isogawa, Tomoyuki Kajiwara, Yuki Arase

Extending a Parser to Distant Domains Using a Few Dozen Partially Annotated Examples (P18-1)

Challenge:	Existing methods for word embedding reconstruction use only local information of subwords and pre-trained word embeds.
Approach:	They propose a global loss function that uses words other than the target word to improve word embedding reconstruction by a factor of 200.
Outcome:	The proposed method reduces the model size of pre-trained word embeddings by a factor of 200 while preserving its quality.

Copied to clipboard

Vidur Joshi, Matthew Peters, Mark Hopkins

Retrofitting Contextualized Word Embeddings with Paraphrases (D19-1)

Challenge:	Statistical parsers are often criticized for their performance outside of the domain they were trained on . we show that word representations reduce the need for domain adaptation when the target domain is syntactically similar to the source domain.
Approach:	They propose a way to adapt a parser to a syntactically similar target domain using partial annotations.
Outcome:	The proposed model increases the accuracy of a parser on the Wall Street Journal by 1.7% over the previous state-of-the-art model.

Copied to clipboard

Weijia Shi, Muhao Chen, Pei Zhou, Kai-Wei Chang

Improving Constituent Representation with Hypertree Neural Networks (2022.naacl-main)

Challenge:	Contextualized word embeddings can be useful for downstream applications, but they can be over-sensitive to contexts.
Approach:	They propose a method to retrofit contextualized word embeddings with paraphrases to minimize the variance of word representations on paraphrased contexts.
Outcome:	The proposed method improves on sentence classification and inference tasks.

Copied to clipboard

Hao Zhou, Gongshen Liu, Kewei Tu

Semantic Linking in Convolutional Neural Networks for Answer Sentence Selection (D18-1)

Challenge:	Existing methods of span representation are based on simple derivations from word representations and do not utilize compositional structures of natural language.
Approach:	They propose a hypertree neural network that is structured with constituency parse trees to improve representations of constituent spans.
Outcome:	The proposed model improves representations of constituent spans using constituency parse trees.

Copied to clipboard

Massimo Nicosia, Alessandro Moschitti

Challenge:	Recent NLP approaches that model relations between text use complex architectures and attention.
Approach:	They propose to use labelled data to model semantic relations between two pieces of text . they use word representations to encode matching features directly in the word representation .
Outcome:	The proposed approach beats tree kernel models and neural models with similar input encodings while keeping the model simple and fast to train.

Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models (2021.naacl-main)

Copied to clipboard

Yuxuan Lai, Yijia Liu, Yansong Feng, Songfang Huang, Dongyan Zhao

Joint Aspect and Polarity Classification for Aspect-based Sentiment Analysis with End-to-End Neural Networks (D18-1)

Challenge:	Pre-trained language models process text as a sequence of characters, ignoring more coarse granularity, e.g., words.
Approach:	They propose a new pre-training paradigm for Chinese that incorporates word representations along with characters and can model a sentence in a multi-granular manner.
Outcome:	The proposed model can bring an average increase of 1.5% under the 12-layer setting, which achieves new state-of-the-art among base-size models on the CLUE benchmarks.

Copied to clipboard

Martin Schmitt, Simon Steinheber, Konrad Schreiber, Benjamin Roth

Sentiment Analysis

Challenge:	a new approach for aspect-based sentiment analysis is proposed . we compare the performance of the proposed approach with pipeline approaches .
Approach:	They propose a model for aspect-based sentiment analysis that uses a convolutional neural network and fasttext embeddings to combine the two approaches.
Outcome:	The proposed model outperforms pipeline approaches in aspects-based sentiment analysis.

Documents Representation via Generalized Coupled Tensor Chain with the Rotation Group constraint (2021.findings-acl)

Copied to clipboard

Igor Vorona, Anh-Huy Phan, Alexander Panchenko, Andrzej Cichocki

Dict-BERT: Enhancing Language Model Pre-training with Dictionary (2022.findings-acl)

Challenge:	despite the diversity of linguistic structures, vector embedding models lack order-preserving properties . current methods for learning linguistic structure can be expensive and time-consuming .
Approach:	They propose a method for embedding documents and words in rotation group . they capture word order and higher-order word interactions .
Outcome:	The proposed model achieves the best results in document classification benchmarks.

Copied to clipboard

Wenhao Yu, Chenguang Zhu, Yuwei Fang, Donghan Yu, Shuohang Wang, Yichong Xu, Michael Zeng, Meng Jiang

Quantifying the morphosyntactic content of Brown Clusters (N19-1)

Challenge:	Pre-trained language models (PLMs) capture word semantics in different contexts, hence the embeddings of rare words on the tail are poorly optimized.
Approach:	They propose to leverage definitions of rare words in dictionaries to enhance language model pre-training by leveraging dictionary definitions.
Outcome:	The proposed model improves understanding of rare words and boosts performance on various NLP downstream tasks.

Copied to clipboard

Manuel R. Ciosici, Leon Derczynski, Ira Assent

Text Clustering

Challenge:	Using corpora representing several language families, we show that word clusters are highly capable at distinguishing Parts of Speech.
Approach:	They propose to use Brown and Exchange word clusters to represent morphosyntactic information in NLP systems.
Outcome:	The proposed clusters are highly capable at distinguishing Parts of Speech and can be used to perform tasks dependent on morphosyntactic information.

Few-Shot Learning Translation from New Languages (2025.emnlp-main)

Copied to clipboard

Carlos Mullov, Alexander Waibel

Multi-Domain Neural Machine Translation with Word-Level Adaptive Layer-wise Domain Mixing (2020.acl-main)

Challenge:	Recent work shows strong transfer learning capability to unseen languages in sequence-to-sequence neural networks . current transfer learning methods require much less downstream task data than would otherwise be required.
Approach:	They first train word embeddings models on varying amounts of data and plug them into a machine translation model.
Outcome:	The proposed model can learn Flores with only 500 parallel sentences and 31,250 sentences of monolingual data, and it can exceed 15 BLEU on unseen languages.

Copied to clipboard

Haoming Jiang, Chen Liang, Chong Wang, Tuo Zhao

Batch IS NOT Heavy: Learning Word Representations From All Samples (P18-1)

Challenge:	Existing multi-domain neural machine translation models lack adaptation to individual domains.
Approach:	They propose a multi-domain neural machine translation model with individual modules for each domain . they use word-level, adaptive and layer-wise domain mixing to achieve this .
Outcome:	The proposed model outperforms existing models in several NMT tasks.

Copied to clipboard

Xin Xin, Fajie Yuan, Xiangnan He, Joemon M. Jose

Improving Named Entity Recognition by Jointly Learning to Disambiguate Morphological Tags (C18-1)

Challenge:	Stochastic Gradient Descent with negative sampling is the most prevalent approach to learn word representations.
Approach:	They propose a method that uses batch gradient learning to generate word representations from all training samples.
Outcome:	The proposed method outperforms sampling-based methods on several benchmark tasks.

Copied to clipboard

Onur Güngör, Suzan Uskudarli, Tunga Güngör

Syntactic Text Processing

Challenge:	Named entity recognition (NER) taggers require external morphological disambiguation tools to function which are hard to obtain or non-existent for many languages.
Approach:	They propose a model which jointly learns NER and MD taggers in languages for which one can provide a list of candidate morphological analyses.
Outcome:	The proposed model can be trained independently of the morphological annotation schemes, and it performs competitively.

Preposition Sense Disambiguation and Representation (D18-1)

Copied to clipboard

Hongyu Gong, Jiaqi Mu, Suma Bhat, Pramod Viswanath

Spot the Odd Man Out: Exploring the Associative Power of Lexical Resources (D18-1)

Challenge:	Prepositions are highly polysemous and their variegated senses encode significant semantic information.
Approach:	They match each preposition’s context and their interplay to the geometry of the word vectors to the left and right of the preposition.
Outcome:	The proposed algorithm is comparable to and better than state-of-the-art on two benchmark datasets.

Copied to clipboard

Gabriel Stanovsky, Mark Hopkins

Fusion: Towards Automated ICD Coding via Feature Compression (2021.findings-acl)

Challenge:	Existing word embeddings assign only one vector to each word, resulting in word disambiguation on smaller scales.
Approach:	They propose a task which aims to test different properties of word representations.
Outcome:	The proposed task is intuitive enough to annotate on a large scale while teasing out properties of popular lexical resources.

Copied to clipboard

Junyu Luo, Cao Xiao, Lucas Glass, Jimeng Sun, Fenglong Ma

Challenge:	Existing methods to assign ICD codes from unstructured clinical notes are noisy and prone to errors.
Approach:	They propose a feature compressed ICD coding model called Fusion to address this problem.
Outcome:	The proposed model outperforms existing models on two widely used datasets.

From Characters to Words: Hierarchical Pre-trained Language Model for Open-vocabulary Language Understanding (2023.acl-long)

Copied to clipboard

Li Sun, Florian Luisier, Kayhan Batmanghelich, Dinei Florencio, Cha Zhang

Automated Concatenation of Embeddings for Structured Prediction (2021.acl-long)

Challenge:	Current models for natural language understanding require a preprocessing step to convert raw text into discrete tokens.
Approach:	They propose a hierarchical open-vocabulary language model that adopts a shallow Transformer architecture to learn word representations from their characters and a deep inter-word Transformer module that contextualizes each word representation by attending to the entire word sequence.
Outcome:	The proposed model outperforms baselines on various downstream tasks and is robust to textual corruption and domain shift.

Copied to clipboard

Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, Kewei Tu

Learning to Generate Word Representations using Subword Information (C18-1)

Challenge:	Recent work shows that better word representations can be obtained by concatenating different types of embeddings.
Approach:	They propose to automate the process of finding better concatenated embeddings for structured prediction tasks by concatending different types of embeddables.
Outcome:	The proposed approach outperforms baselines and achieves state-of-the-art with fine-tuned embeddings on 6 tasks and 21 datasets.

Copied to clipboard

Yeachan Kim, Kang-Min Kim, Ji-Min Lee, SangKeun Lee

A Deterministic Algorithm for Bridging Anaphora Resolution (D18-1)

Challenge:	Existing word-based approaches to learning word representations are blind to subword information in words.
Approach:	They propose a character-based word representation approach to learn word representations from characters.
Outcome:	The proposed model outperforms baseline models that regard words as atomic units . the proposed model achieves 18.5% improvement on average in perplexity for morphologically rich languages .

Copied to clipboard

Yufang Hou

On the Importance of Distinguishing Word Meaning Representations: A Case Study on Reverse Dictionary Mapping (N19-1)

Challenge:	Existing methods for bridging anaphora resolution only consider NPs’ head nouns and thus do not capture the semantics of NP.
Approach:	They propose a deterministic approach to bridging anaphora resolution which represents the semantics of an NP based on its head noun and modifications.
Outcome:	The proposed approach achieves competitive results compared to the best system in Hou et al. (2013) which explores Markov Logic Networks to model the problem.

Copied to clipboard

Mohammad Taher Pilehvar

ProFormer: Towards On-Device LSH Projection Based Transformers (2021.eacl-main)

Challenge:	Sense representations target meaning conflation deficiency but their potential impact has not been investigated in downstream NLP applications.
Approach:	They propose to use a reverse dictionary system to address meaning conflation deficiency . they propose to integrate senses into the system to improve semantic understanding .
Outcome:	The proposed approach can improve the performance of a downstream NLP application.

Copied to clipboard

Chinnadhurai Sankar, Sujith Ravi, Zornitsa Kozareva

Constituency Parsing with a Self-Attentive Encoder (P18-1)

Challenge:	ProFormer is a projection based transformer architecture that is faster and lighter making it suitable to deploy to memory constraint devices such as mobile phones, watches and IoT.
Approach:	They propose a projection based transformer architecture that generates word representations on-the-fly without embedding lookup tables and a local projection attention layer that transforms the input sequence of N LSH word projections into a sequence of K representations.
Outcome:	The proposed architecture reduces memory footprint from 92.16 MB to 1.7 KB and requires 16x less computation overhead making it suitable to deploy to memory constraint devices and preserve user privacy.

Copied to clipboard

Nikita Kitaev, Dan Klein

Combining Static Word Embeddings and Contextual Representations for Bilingual Lexicon Induction (2021.findings-acl)

Challenge:	Recent work on LSTM encoders based on recurrent neural networks has led to improvements in constituency parsing accuracy.
Approach:	They propose to replace an LSTM encoder with a self-attentive architecture to improve a discriminative constituency parser.
Outcome:	The proposed model outperforms the previous best-published results on 8 of the 9 languages in the SPMRL dataset.

Copied to clipboard

Jinpeng Zhang, Baijun Ji, Nini Xiao, Xiangyu Duan, Min Zhang, Yangbin Shi, Weihua Luo

Definition Frames: Using Definitions for Hybrid Concept Representations (2020.coling-main)

Challenge:	Bilingual Lexicon Induction (BLI) aims to map words in one language to their translations in another.
Approach:	They propose a mechanism to combine static word embeddings and contextual representations to utilize the advantages of both paradigms.
Outcome:	The proposed method improves performance on supervised and unsupervised BLI benchmarks on all language pairs by average improving 3.2 points over baselines.

Copied to clipboard

Evangelia Spiliopoulou, Artidoro Pagnoni, Eduard Hovy

Explainability & Interpretability in NLP

Challenge:	a new hybrid representation is proposed that encodes semantic information extracted from definitions.
Approach:	They propose a matrix distributed representation extracted from definitions where each dimension is semantically interpretable.
Outcome:	The proposed representations have competitive performance with other distributional semantic approaches on word similarity tasks.

Effect of Post-processing on Contextualized Word Representations (2022.coling-1)

Copied to clipboard

Hassan Sajjad, Firoj Alam, Fahim Dalvi, Nadir Durrani

Decomposing Co-occurrence Matrices into Interpretable Components as Formal Concepts (2024.findings-acl)

Challenge:	Post-processing of static embeddings has been shown to improve their performance on both lexical and sequence-level tasks.
Approach:	They standardize individual neuron activations using z-score, min-max normalization, and remove top principal components using the all-but-the-top method.
Outcome:	The proposed method unwraps vital information present in the representations for both lexical and sequence classification tasks.

Copied to clipboard

Akihiro Maeda, Takuma Torii, Shohei Hidaka

Explainability & Interpretability in NLP

Challenge:	Existing studies have shown that word vectors capture relational meanings, but the interpretability of their dimensions remains an open issue.
Approach:	They employ the mathematical methodology of Formal Concept Analysis to examine word embeddings using a count-based co-occurrence matrix.
Outcome:	The proposed model shows that the formal concepts identified align with interpretable categories, as shown in the category completion task.

Shrinking Japanese Morphological Analyzers With Neural Networks and Semi-supervised Learning (N19-1)

Copied to clipboard

Arseny Tolmachev, Daisuke Kawahara, Sadao Kurohashi

Syntactic Text Processing

Challenge:	Modern neural morphological analyzers consume gigabytes of memory.
Approach:	They propose a method which uses unigram character embeddings to train a model on labels produced by a state-of-the-art analyzer.
Outcome:	The proposed model outperforms dictionary-based methods in Japanese and Chinese . it uses less than 15 megabytes of space and is much smaller than the dictionary- based one .

Wikipedia Entities as Rendezvous across Languages: Grounding Multilingual Language Models by Predicting Wikipedia Hyperlinks (2021.naacl-main)

Copied to clipboard

Iacer Calixto, Alessandro Raganato, Tommaso Pasini

Word-Level Loss Extensions for Neural Temporal Relation Classification (C18-1)

Challenge:	Masked language models have become the de facto standard when processing text . however, these models are evaluated in a monolingual setting only .
Approach:	They propose a language-independent entity prediction task as an intermediate training procedure to ground word representations on entity semantics and bridge the gap between different languages.
Outcome:	The proposed approach bridges the gap between word representations and knowledge graphs by using a shared vocabulary of entities.

Copied to clipboard

Artuur Leeuwenberg, Marie-Francine Moens

A Regularization Approach for Incorporating Event Knowledge and Coreference Relations into Neural Discourse Parsing (D19-1)

Challenge:	Unsupervised pre-trained word embeddings are used for many tasks in natural language processing to leverage unlabeled textual data.
Approach:	They extend the model's task loss with an unsupervised auxiliary loss on the word-embedding level of the model to ensure that the learned word representations contain both task-specific features and more general features.
Outcome:	The proposed model improves on the task of extracting narrative containment relations from clinical records using a general-domain part-of-speech tagger as linguistic resource.

Copied to clipboard

Zeyu Dai, Ruihong Huang

Manifold Learning-based Word Representation Refinement Incorporating Global and Local Information (2020.coling-main)

Challenge:	Existing approaches to discourse parsing use commonsense knowledge and linguistic constraints to integrate them into neural network models.
Approach:	They propose a knowledge regularization approach that integrates linguistic constraints with contexts for deriving word representations.
Outcome:	The proposed approach outperforms previous systems on the benchmark dataset PDTB for discourse parsing.

Copied to clipboard

Wenyu Zhao, Dong Zhou, Lin Li, Jinjun Chen

Accelerated High-Quality Mutual-Information Based Word Clustering (2020.lrec-1)

Challenge:	Recent studies show word embedding models underestimate similarities between similar words and overestimate similarities between distant words.
Approach:	They propose two new word embedding methods that align original and re-fined embeddable spaces to a new refined semantic space.
Outcome:	The proposed methods outperform state-of-the-art methods for word representation refinement.

Copied to clipboard

Manuel R. Ciosici, Ira Assent, Leon Derczynski

Text Clustering

Challenge:	Word clustering is a hard hierarchical clustering that uses short-range distributional information to construct clusters.
Approach:	They propose to use a hierarchical clustering algorithm with a fixed-width beam to build clusters that outperform other word representations.
Outcome:	The proposed method outperforms the original methods in the computation of hierarchical and flat clusters.

Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal Transformer (2020.acl-main)

Copied to clipboard

Jianfei Yu, Jing Jiang, Li Yang, Rui Xia

Incorporating Syntactic and Semantic Information in Word Embeddings using Graph Convolutional Networks (P19-1)

Challenge:	Existing methods for named entity recognition ignore visual context bias . NER is a key component of many information extraction tasks .
Approach:	They propose to use a multimodal interaction module to generate word-aware visual representations and leverage purely text-based entity span detection as an auxiliary module to guide the final predictions.
Outcome:	The proposed approach achieves state-of-the-art on two benchmark datasets.

Copied to clipboard

Shikhar Vashishth, Manik Bhandari, Prateek Yadav, Piyush Rai, Chiranjib Bhattacharyya, Partha Talukdar

Xiao Zhang, Ji Wu, Dejing Dou

Challenge:	Existing word embedding methods utilize sequential context of a word to learn its embeddment, but such methods result in an explosion of the vocabulary size.
Approach:	They propose a flexible Graph Convolution based method for learning word embeddings that utilizes the dependency context of a word without increasing the vocabulary size.
Outcome:	The proposed model outperforms existing methods on intrinsic and extrinsic tasks and provides an advantage when used with ELMo.

Delta Embedding Learning (P19-1)

Copied to clipboard

Graph Convolutions over Constituent Trees for Syntax-Aware Semantic Role Labeling (2020.emnlp-main)

Challenge:	Unsupervised word embeddings have limitations to the semantics of words and inadequate fine-tuning of embedded word can lead to suboptimal performance.
Approach:	They propose a method that optimizes word embeddings by regularizing them incrementally to ensure they are tuned in an incremental way.
Outcome:	The proposed method improves performance on various NLP tasks and shows that it absorbs semantic information without "forging"

Copied to clipboard

Diego Marcheggiani, Ivan Titov

Massive vs. Curated Embeddings for Low-Resourced Languages: the Case of Yorùbá and Twi (2020.lrec-1)

Challenge:	Semantic role labeling (SRL) is the task of identifying predicates and labeling argument spans with semantic roles.
Approach:	They propose to use graph convolutional networks to encode constituents and inform an SRL system by combining word representations of the first and last words in a constituent tree.
Outcome:	The proposed model is compared with other models and shows that it is more efficient than dependency trees.

Copied to clipboard

Jesujoba Alabi, Kwabena Amponsah-Kaakyire, David Adelani, Cristina España-Bonet

Transferable Neural Projection Representations (N19-1)

Challenge:	a recent study shows that word embeddings can be useful for training downstream natural language processing tasks.
Approach:	They compare word embeddings obtained by word embeds from curated corpora with a language-dependent processing.
Outcome:	The proposed model compares word embeddings with word embeds from curated corpora and a language-dependent processing on two African languages.

Copied to clipboard

Chinnadhurai Sankar, Sujith Ravi, Zornitsa Kozareva

Domain adaptation for part-of-speech tagging of noisy user-generated text (N19-1)

Challenge:	Neural word embeddings require lookup and a large memory footprint making it hard to deploy on-device.
Approach:	They propose a skip-gram based architecture coupled with Locality-Sensitive Hashing projections to learn efficient dynamically computable representations.
Outcome:	The proposed model performs better than previous models on multiple NLP tasks.

Copied to clipboard

Luisa März, Dietrich Trautmann, Benjamin Roth

One Word, Two Sides: Traces of Stance in Contextualized Word Representations (2022.coling-1)

Challenge:	Existing POS taggers for canonical German text achieve good results around 97% accuracy, but when applying these trained models to out-of-domain data the performance decreases drastically.
Approach:	They propose a neural network that trains an out-of-domain model on a large newswire corpus and transfers those weights by using them as a prior for a model trained on the target domain.
Outcome:	The proposed model achieves a tagging accuracy of slightly over 90%, improving on the previous state of the art for this task.

Copied to clipboard

Aina Garí Soler, Matthieu Labeau, Chloé Clavel

Part-of-Speech Tagging for Code-Switched, Transliterated Texts without Explicit Language Identification (D18-1)

Challenge:	a Lexical Semantic Change study examines the way we use words . it focuses on the use of words by people who disagree on a particular topic .
Approach:	They examine whether word embeddings reflect the way we use words . they use BERT embeddables from datasets with stance annotations to examine this question .
Outcome:	The results show that people with opposing stances use different words when talking about a topic . the results are not related to studies that investigate the usage of specific words across different viewpoints.

Copied to clipboard

Kelsey Ball, Dan Garrette

Tagging

Challenge:	Code-switching is a challenge for NLP due to the lack of representative data for training models.
Approach:	They propose a model that is trained exclusively on monolingual resources but can be applied to unseen code-switched text at inference time.
Outcome:	The proposed model outperforms standard models on Hindi-English part-of-speech tagging and on unannotated code-switched text with alternate scripts.

Don’t Just Scratch the Surface: Enhancing Word Representations for Korean with Hanja (D19-1)

Copied to clipboard

Kang Min Yoo, Taeuk Kim, Sang-goo Lee

Decoupled Vocabulary Learning Enables Zero-Shot Translation from Unseen Languages (2024.acl-long)

Challenge:	Existing knowledge of Korean and Chinese is based on cultural and historical reasons.
Approach:	They propose a method for improving Korean word representations using additional linguistic annotation by leveraging the fact that Hanja is closely related to Chinese.
Outcome:	The proposed approach improves representations on a novel Korean news headline generation task.

Copied to clipboard

Carlos Mullov, Quan Pham, Alexander Waibel

CA-EHN: Commonsense Analogy from E-HowNet (2020.lrec-1)

Challenge:	Multilingual neural machine translation systems learn to map sentences of different languages into a common representation space.
Approach:	They propose a setup where we decouple learning of vocabulary and syntax and train to translate while keeping those word representations frozen.
Outcome:	The proposed setup achieves near parity with a supervised setting on the TED domain with varying number of languages seen by the encoder.

Copied to clipboard

Peng-Hsuan Li, Tsan-Yu Yang, Wei-Yun Ma

Reasoning

Challenge:	Existing word analogy datasets rely on handcrafted words with only dozens of predefined relations.
Approach:	They present a commonsense word analogy dataset with 90,505 analogies . they use an ontology that annotates 88K Chinese words with their structured sense definitions and English translations.
Outcome:	The proposed dataset shows that word representations embed commonsense knowledge.

Aligning Multilingual Embeddings for Improved Code-switched Natural Language Understanding (2022.coling-1)

Copied to clipboard

Barah Fazili, Preethi Jyothi

Improving a Neural-based Tagger for Multiword Expressions Identification (L18-1)

Challenge:	a recent study has shown that multilingual models can be effective on monolingual data but need additional training to work well with code-switched text.
Approach:	They propose to train multilingual models with alignment objectives using parallel text . they find such an explicit alignment step improves performance on code-switched NLP tasks .
Outcome:	The proposed model improves on Hindi-English Sentiment Analysis, Named Entity Recognition and Question Answering tasks.

Copied to clipboard

Dušan Variš, Natalia Klyueva

A Structural Probe for Finding Syntax in Word Representations (N19-1)

Challenge:	MUMULS tagger for automatic detection of verbal multiword expressions is based on neural networks . character-level embeddings can improve the performance, reducing out-of-vocabulary rate . multiword Expressions are viewed by computational linguists as a "pain in the neck of NLP"
Approach:	They propose to improve MUMULS, a tagger for automatic detection of verbal multiword expressions.
Outcome:	The proposed tagger performed better on Czech language than the previous taggers.

Copied to clipboard

John Hewitt, Christopher D. Manning

Precisely the Point: Adversarial Augmentations for Faithful and Informative Text Generation (2022.emnlp-main)

Challenge:	Existing methods for detecting syntactic knowledge do not test whether syntax trees are embedded in a linear transformation of a neural network’s word representation space.
Approach:	They propose a structural probe which evaluates whether syntax trees are embedded in a linear transformation of a neural network’s word representation space.
Outcome:	The proposed model shows that entire syntax trees are embedded in deep models’ vector geometry.

Copied to clipboard

Wenhao Wu, Wei Li, Jiachen Liu, Xinyan Xiao, Sujian Li, Yajuan Lyu

Robustness in NLP

Challenge:	Existing models for text generation are weak enough to handle perturbations in inputs, leading to degeneration in faithfulness and informativeness.
Approach:	They propose a framework for improving faithfulness and informativeness of Seq2Seq models by perturbing word representations and word swapping.
Outcome:	The proposed framework improves faithfulness and informativeness of Seq2Seq models under automatic and human evaluation settings.

Signal in Noise: Exploring Meaning Encoded in Random Character Sequences with Character-Aware Language Models (2022.acl-long)

Copied to clipboard

Mark Chu, Bhargav Srinivasa Desikan, Ethan Nadler, Donald Ruggiero Lo Sardo, Elise Darragh-Ford, Douglas Guilbeault

mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models (2022.acl-long)

Challenge:	Existing words represent an extremely small fraction of the space of possible character level n-grams (word forms) yet, a plethora of insights into language learning have emerged from inquiries into language beyond extant words, such as the grammatical errors and inference patterns children exhibit when distinguishing extant word from non-linguistic auditory signals.
Approach:	They propose that random character n-grams provide a novel context for studying word meaning both within and beyond extant language.
Outcome:	The proposed model identifies an axis in its high-dimensional embedding space that separates these classes of n-grams from other classes of characters and relates to structure within extant language, including word part-of-speech, morphology, and concept concreteness.

Copied to clipboard

Ryokan Ri, Ikuya Yamada, Yoshimasa Tsuruoka

Learning Word Representations with Cross-Sentence Dependency for End-to-End Co-reference Resolution (D18-1)

Challenge:	Existing methods for improving multilingual models only use entity information in pretraining and do not explicitly use entities in downstream tasks.
Approach:	They propose to leverage Wikipedia entity representations for downstream tasks . they train a multilingual language model with 24 languages with entity representation .
Outcome:	The proposed model outperforms word-based models in cross-lingual transfer tasks.

Copied to clipboard

Hongyin Luo, Jim Glass

A Prism Module for Semantic Disentanglement in Name Entity Recognition (P19-1)

Challenge:	Existing word embedding models generate word representations by running long short-term memory recurrent neural networks on each sentence of an input article or conversation separately.
Approach:	They propose a word embedding model that learns cross-sentence dependency . they use linear sentence linking and attentional sentence linking to learn cross-entry dependency based on context sentences .
Outcome:	The proposed model improves end-to-end co-reference resolution by taking knowledge from context sentences and the entire document.

Copied to clipboard

Kun Liu, Shen Li, Daqi Zheng, Zhengdong Lu, Sheng Gao, Si Li

Linguistically-Informed Self-Attention for Semantic Role Labeling (D18-1)

Challenge:	Xu et al., 2015) proposed a noise reduction mechanism to disentangle semantics of words . hard and soft attention mechanisms are used to reduce noise in NLP tasks .
Approach:	They propose a prism module to disentangle semantic aspects of words and reduce noise . they propose combining prism modules with downstream models to improve model performance .
Outcome:	The proposed method significantly improves the performance of baselines on named entity recognition (NER) tasks.

Copied to clipboard

Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, Andrew McCallum

Type-dependent Prompt CycleQAG : Cycle Consistency for Multi-hop Question Generation (2022.coling-1)

Challenge:	Existing models of semantic role labeling use no explicit linguistic features. prior work has shown that syntax trees can dramatically improve SRL decoding.
Approach:	They propose a neural network model that incorporates syntax using only raw tokens . they show that LISA out-performs the state-of-the-art with contextually-encoded word representations a 1.0 F1 on newswire and 2.0 F1 in out-of domain text .
Outcome:	The proposed model outperforms the state-of-the-art model with word embeddings and predicted predicates.

Copied to clipboard

Seungyeon Lee, Minho Lee

Low-Resource NLP

Challenge:	Existing research on multi-hop question generation (QG) has not been done due to its complexity.
Approach:	They propose a type-dependent prompt cycleQAG with a cycle consistency loss . they propose to use the question type and words related to the correct answer as prompts .
Outcome:	The proposed model outperforms the baseline model by 10.38% based on ROUGE score.

Learning Word Vectors for 157 Languages (L18-1)

Copied to clipboard

Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, Tomas Mikolov

Multi-Granularity Interaction Network for Extractive and Abstractive Multi-Document Summarization (2020.acl-main)

Challenge:	Distributed word representations, or word vectors, have been used in natural language processing for many tasks.
Approach:	They propose to use the encyclopedia Wikipedia and the common crawl corpus to train distributed word representations on large corpora and use them in downstream tasks.
Outcome:	The proposed model performs very well on 10 languages for which evaluation dataset exists.

Copied to clipboard

Hanqi Jin, Tianming Wang, Xiaojun Wan

GGP: Glossary Guided Post-processing for Word Embedding Learning (2020.lrec-1)

Challenge:	Existing methods for document summarization use extractive and abstractive representations, but they don't take into account hierarchical structure of document clusters.
Approach:	They propose a multi-granularity interaction network for extractive and abstractive multi-document summarization which jointly learn semantic representations for words, sentences, and documents.
Outcome:	The proposed model outperforms baseline methods and achieves the best results on the Multi-News dataset.

Copied to clipboard

Ruosong Yang, Jiannong Cao, Zhiyuan Wen

Learn Your Tokens: Word-Pooled Tokenization for Language Modeling (2023.findings-emnlp)

Challenge:	Existing word embedding models require much training time and domain knowledge to improve.
Approach:	They propose a GGP-based word embedding model that incorporates the glossary and learns sense representations.
Outcome:	The proposed model outperforms existing models on topical/functional similarity datasets by 4.1% and 7%.

Copied to clipboard

Avijit Thawani, Saurabh Ghanekar, Xiaoyuan Zhu, Jay Pujara

Sequential Modelling of the Evolution of Word Representations for Semantic Change Detection (2020.emnlp-main)

Challenge:	Language models typically tokenize text into subwords, using a deterministic, hand-engineered heuristic of combining characters into longer surface-level strings such as ‘ing’ or whole words.
Approach:	They propose a 'learn your tokens' scheme which pooles bytes/characters into word representations and decodes individual characters/bytes per word in parallel.
Outcome:	The proposed tokenizer outperforms subword models and byte/character models over the word boundary and outperformed on rare words by a factor of 30!

Copied to clipboard

Adam Tsakalidis, Maria Liakata

Contextualized Semantic Distance between Highly Overlapped Texts (2023.findings-acl)

Challenge:	Existing models that detect semantically shifted words do not account for its evolution through time.
Approach:	They propose three variants of sequential models for detecting semantically shifted words . they demonstrate that temporal modelling of word representations yields a clear-cut advantage .
Outcome:	The proposed models account for the changes in word representations over time.

Copied to clipboard

Letian Peng, Zuchao Li, Hai Zhao

An Empirical Study on Leveraging Position Embeddings for Target-oriented Opinion Words Extraction (2021.emnlp-main)

Challenge:	Conventional semantic metrics are based on word representations and are vulnerable to disturbance of overlapped components with similar representations.
Approach:	They propose a mask-and-predict strategy to evaluate the semantic distance between the overlapped sentences using words in the longest common sequence as neighboring words and use masked language modeling to predict their positions.
Outcome:	The proposed method outperforms the state-of-the-art in domain adaption by a huge margin.

Copied to clipboard

Samuel Mensah, Kai Sun, Nikolaos Aletras

VROAV: Using Iconicity to Visually Represent Abstract Verbs (2020.lrec-1)

Challenge:	Current methods for extracting opinion words for an aspect in text leverage position embeddings to capture relative position of word to the target.
Approach:	They propose to use pretrained word embeddings to extract opinion words for a given aspect in text.
Outcome:	The proposed methods outperform current methods on a task based on pre-trained word embeddings and position embedders.

Copied to clipboard

Simone Scicluna, Carlo Strapparava

Multimodality

Challenge:	Visual languages like sign languages reveal enlightening patterns across signs of similar meanings, pointing towards the possibility of identifying clusters of iconic meanings.
Approach:	a new verb classification system is proposed to visually represent 20 classes of abstract verbs.
Outcome:	The proposed system could be used as a language learning aid or as linguistic comprehension tool for digital text.

Time-Aware Language Modeling for Historical Text Dating (2023.findings-emnlp)

Copied to clipboard

Han Ren, Hai Wang, Yajie Zhao, Yafeng Ren

Linking Adaptive Structure Induction and Neuron Filtering: A Spectral Perspective for Aspect-based Sentiment Analysis (2024.lrec-main)

Challenge:	Existing approaches to automatic text dating ignore diachronic change of words, which may affect the efforts of text modeling.
Approach:	They propose a time-aware language model to learn temporal word representations by transferring language models of general domains to those of time-specific ones and build a hierarchical modeling approach to represent diachronic documents by encoding them with temporal representations.
Outcome:	The proposed model outperforms state-of-the-art approaches in historical text dating and other NLP tasks.

Copied to clipboard

Hao Niu, Maoyi Wang, Yun Xiong, Biao Yang, Xing Jia, Zhonglei Guo

Sentiment Analysis

Challenge:	incorporating structure information can improve the performance of aspect-based sentiment analysis.
Approach:	They propose a method to conduct neuron-level manipulations on word representations in the frequency domain.
Outcome:	The proposed method can achieve or come close to state-of-the-art in ABSA.

ScanDL: A Diffusion Model for Generating Synthetic Scanpaths on Texts (2023.emnlp-main)

Copied to clipboard

Lena Bolliger, David Reich, Patrick Haller, Deborah Jakobi, Paul Prasse, Lena Jäger

Challenge:	Eye movements in reading are a key part of psycholinguistic research, but the lack of eye movement data and its unavailability at application time pose a major challenge for this line of research.
Approach:	They propose a novel sequence-to-sequence diffusion model that generates synthetic scanpaths on texts by leveraging pre-trained word representations and jointly embedding both the stimulus text and the fixation sequence.
Outcome:	The proposed model outperforms state-of-the-art models in psycholinguistic analysis and is able to exhibit human-like reading behavior.

Better Embeddings with Coupled Adam (2025.acl-long)

Copied to clipboard

Felix Stollenwerk, Tobias Stollenwerk